Open-domain sketch understanding: The nuSketch approach

Sketching is often used when working out ideas. This combination of drawing and conceptual labeling is a very natural and effective form of communication and problem-solving. Creating software that can participate in sketching provides many challenges. We outline the nuSketch approach to sketch understanding, which focuses on visual and conceptual understanding instead of recognition. We summarize three experiments in progress with the sketching Knowledge Entry Associate (sKEA), the first open-domain sketch understanding system. sKEA utilizes a number of visual and qualitative spatial reasoning capabilities and human-like analogical matching to tackle a variety of tasks. We summarize some experimental results and outline future plans.